1. Field of the Invention
The present invention relates generally to a picture inputting apparatus for use in picture input to an animation processing system, etc., and more particularly to an image inputting apparatus using a high-resolution solid state image pickup device to acquire low-resolution whole pictures and high-resolution partial pictures.
2. Description of the Related Art
Up until now, in order to acquire detailed picture information while securing a wide range of observation range, picture inputting apparatuses have employed a method in which the resolutions are uniformly improved by raising the mounting density of image pickup devices such as CCD image pickup devices or CMOS image pickup devices. The raised resolutions of the image pickup devices lead to securement of sufficient resolutions and to acquisition of detailed contents for a relatively small target as well even though a wide-angle optical lens is used to shoot a wider area.
However, such a uniform raise of the resolutions of the image pickup devices has disadvantageously resulted in enormous amount of information possessed by the pictures and in a lot of time required to transmit the pictures since limitation is imposed on the capacities of the transmission path for transmitting the pictures and on the input capacities of the apparatuses for inputting pictures into the image processor or a display unit, thus leading to an extreme lowering of the frame rate by which pictures can be obtained.
When using an image pickup device of e.g., 3000000 pixels of 1700×1700 pixels in length and width, the frame rate to be acquired is of the order of 8 fps (frames per second). For this reason, if a high-resolution image pickup device is used for image pickup when the target is moving, it has been difficult to follow a target whose position varies with time and to use it as the image inputting apparatus of the time-varying image processing system.
Proposed as the prior art for preventing such a lowering of the frame rate is a method for realizing the simultaneous acquisition of the whole observation and the partial detailed pictures while suppressing the total amount of information, by acquiring, for the whole picture of the target to be shot, pictures with reduced pixels through the lowering of the resolution and simultaneously by acquiring, for only a part of the wholeness, detailed pictures without increasing the pixel count through the shooting of high-resolution pictures without lowering the resolution.
To realize this, there are roughly two approaches, first is an approach using a device, and second is an approach using a plurality of image pickup sensors. The former is a method as proposed in Japanese Patent Laid-open (kokai) Pub. No. H9-214836. For shooting of the wholeness of the target, a high-resolution image pickup device is used which consists of a plurality of photo-receptive cells (photo-receptive pixels) which are arranged at a high density, and the obtained high-resolution pictures are thinned out for accumulation into a memory in the form of a low-resolution whole picture. On the contrary, for only a part of the whole picture, the original high-density picture information is extracted to accumulate it as a detailed picture into another memory, with the two pictures being extracted switchingly every one frame or for each more than one frames such that the two pictures accumulated in their respective memories are synthesized into a single picture, which in turn is output in NTSC format so that the picture can be monitored by the NTSC television monitor.
For the camera system using a high-resolution image pickup device, Japanese Patent Laid-open (kokai) Pub. No. 2000-32318 proposes a method for simultaneously acquiring a whole picture for field angle registration and a partial detailed picture for auto-focusing, for a picture within the same frame. The summary of the realization mean is as follows.
Each row of one frame picture is scanned in sequence. The rows required for only the low-resolution picture of the whole picture are subjected to thinning-out processing for each row and, for the rows requiring it, the results of execution of the thinning-out processing on the pixel columns within the rows are accumulated in a memory for thinned-out pictures.
In the rows to acquire a detailed picture, information on all pixels within the row is accumulated in another memory for detailed pictures without performing the thinning out processing. The thinning out processing in the row and column directions are effected on the partial pictures unsubjected to the thinning out processing accumulated in the memory for detailed pictures, to synthesize the obtained thinned-out low-resolution pictures of the partial pictures and the low-resolution pictures of remaining portions obtained in the previous stage and accumulated in the memory for thinned-out pictures, to thereby output a low-resolution whole picture.
As to the partial pictures lying in the memory for detailed pictures, original high-resolution pictures for the wholeness of the row are accumulated, and hence portions required as the detailed pictures are extracted for the output as the partial detailed pictures.
The methods proposed in the Patent Documents 1 and 2 allows suppression of the amount of information to be transferred as well as securement of the picture frame rate since the picture size of the whole picture and of the detailed pictures is sufficiently small in terms of the total resolution.
The inventors of the present application have proposed the following as the method using a plurality of image pickup sensors (Japanese Patent Application No. 2001-140925). This method obtains partial detailed pictures by dividing a target projected image into two images by a bisecting optical system, reducing the wholeness of one image using a reducing optical system to allow a single image pickup device to shoot a whole picture, and shooting a part of the other image enlarged by an enlarging optical system using another image pickup device.
The above method is characterized in that at that time, the position to shoot a partial image can move to any position in the whole picture by attaching the image pickup device onto a mechanism capable of varying the position on a plane such as an XY stage. According to this method, it is possible to equivalently obtain 100 times higher resolution by shooting partial pictures among images whose sides are enlarged to 10 times. In addition, it is possible to restrict the amount of information to the doubled VGA amount of information at most by setting the size of the image pickup device shooting the whole picture and the detailed pictures to the size, e.g., of the order of VGA (640×480 pixels).
However, the methods of Patent Documents 1 and 2 have been effected for the purpose of allowing a person to simply perform the field angle adjustment and focusing adjustment in the camera shooting system. The above methods are arranged such that output is made so that synthesized pictures of the whole picture for field angle adjustment and the partial detailed pictures for focusing adjustment appear at the same time in order to allow the person to perform visual adjustments by the NTSC television monitor or such that auto-focusing adjustment is made by detailed pictures of a certain portion while allowing the person to visually observe the whole picture at all times through the NTSC television monitor.
To this end, in the method of Japanese Patent Laid-open (kokai) Pub. No. H9-214836, the whole picture and the partial detailed pictures are embedded in a single NTSC picture (whose resolution is of the order of 512×480 pixels in length and width), and hence the resolutions possessed by the respective pictures become halved. Thus, it is essential for a person to intervene since the portions to be image picked up as detailed pictures must be predefined fixed portions or must be directly designated from the outside of the apparatus due to the lowered amount of information as well as due to independent outputs of the whole picture and the partial detailed pictures.
In the method of Japanese Patent Laid-open (kokai) Pub. No. 2000-32318, the low-resolution picture of the whole scene is merely converted by D/A conversion into analog television signals such as NTSC and is output for field angle verifications, whereas the partial detailed pictures are not output to the outside which are merely used within the interior of the apparatus for auto adjustment of the focus distance. Moreover, the acquiring positions of the partial detailed pictures are predetermined positions or portions designated by a person from the outside of the apparatus.
In case of considering it as a picture inputting apparatus in the time-varying image processing system, it is necessary to input picture information which is sole input information as much as possible, i.e., to input detailed picture information in terms of the stabilization of the processing and a higher function. In view of this point, it is impossible for the camera system which outputs the whole picture and the partial detailed pictures as a single picture to implement desired performances due to its lowered amount of information.
The time-varying image processing system treats scenes which move (vary) with time as processing targets and the required information also varies with time, whereupon it is necessary to automatically set points at which the detailed information is acquired, depending on the contents appearing in the scene which varies with time.
In the conventional apparatuses of Patent Documents 1 and 2, however, the positions to acquire the detailed pictures must manually be designated from the outside of the apparatus. Since the apparatuses do not have any function for varying the position to acquire the detailed pictures depending on the picture information, it would be difficult to meet the requirements for the time-varying image processing system.
Otherwise, the conventional apparatuses suffer another problem on means for creating the whole low-resolution pictures. The conventional apparatuses create the low-resolution picture by merely thinning out the high-resolution pictures. If pixels are thinned out such that the dimensions of the low-resolution picture becomes 1/N of the dimensions of the original high-resolution pictures, the input picture information results in having a N-times higher frequency than the sampling rate since the frequency band possessed by the pictures is unvaried in spite of lowering of the picture sampling frequency to 1/N.
For this reason, aliasing occur at high-frequency portions due to the sampling theorem, resulting in false pictures. If false information is included in the pictures input to the time-varying image processing system, the system may experience mis-recognition and thus abnormal actions. Since such aliasing-based false information cannot be removed in the latter processings, it is inevitable to remove the false information at the stage of conversion into the low-resolution pictures.
In Japanese Patent Application No. 2001-140925, the image pickup device obtaining partial high-resolution picture is moved through mechanical operations by use of, e.g., an XY stage. Due to the weight of the drive of the stage and to the weight of the image pickup device itself, however, it is difficult to keep a high moving speed enough to move from end to end of the picture within the frame rate (e.g., 1/30 sec), and consequently, to acquire detailed pictures at quite different points for each frame rate.
If N frames are required to complete the shooting of any point, when the target is an object which moves at a high speed, the position of the target will offset from the position designated to acquire a detailed picture due to the movement of the target during N frames. It is thus difficult to catch the target needing the detailed picture at the center of the detailed picture, and disadvantageously in case of particularly high speed, the target may possibly deviate from the detailed picture. There arises also a problem that it is impossible to shoot, while varying the shooting positions on a frame rate basis, targets to be shot scattering in a shooting range.